Guidelines for Composing Graphical Interfaces for Fast and Correct Perception of System State: Data Analysis

Load data

Data was recorded with a dedicated software that showed the situations for the different designs. Data of all participants was afterwards concatenated into one file. Meta information about anonymized subjects was recorded in separated file. Load data and merge with subject meta data:

# read dataset
data <- read.csv("DataKonectGuidelines_allSubjects.csv", header = T, sep = ",",stringsAsFactors=FALSE)
# read subject information
subjects <- read.csv("SubjectsKonectGuidelines.csv", header = T, sep = ",",stringsAsFactors=FALSE)
#join data with meta information about subjects. VPID is the identifier for subjects.
data <- merge(data, subjects, by="VPID")
#For simplification in the paper we use IDs "D1" - "D6" for the design concepts, instead of the HMI designer group ID, that were used in the workshops
data$BaseDesign <- mapvalues(data$BaseDesign, from=c('P1P2','P3P4','P5P6','P7P8','P9P10','P13P14') , to=c('D1','D2','D3','D4','D5','D6'))
#We use more readable situation desciptors 
data$DesignState <- mapvalues(data$DesignState, from=c('ok', 'lowSpeed', 'criticalTrim', 'relevantDuK', 'strongWind') , to=c('non-critical','critical ship speed','critical trim state','critical (low) depth under keel','critical wind condition'))

Data selection.

Dataset contains a trace of all events in the recording software. Reaction times and correctness of responses are recorded in datasample with “action = REACTION”

data <- filter(data, action == 'REACTION')

Dataset is from a larger experiment adressing multiple issues. For analysing the effect of guidelines on reaction times and accuracy only data from subjects that conducted A-B-A-B blocks are of interest.

data <- filter(data, experimentScenario == 'ABAB')

Block A and B were iterated twice. To eliminate strong training effects we only analyse data from the second iteration

data <- filter(data, BlockRepetition == 2)

Cutoffs for reaction times: We consider reaction times over 10 seconds as failure. Reaction times include perception, decision and motor response. It is unlikely that this happens in less than 300 ms, which we consider a click-error. Reaction times exceeding these limits were excluded

excluded <- filter(data, reaction_ms > 10000 | reaction_ms < 300)
# number of excluded samples
nrow(excluded)

[1] 5

# exclude according to cutoff times
dsta <- filter(data, reaction_ms <= 10000 & reaction_ms >= 300)

Relevant information for subsequent analysis are * VPID: Identifier for the subject from whom this data sample is, * critical: criticality of shown situation, * reaction_ms: reaction time in ms, * Guideline: whether the currently shown situation is from an initial design (Guideline==0) or from a design after applying the guidelines (Guideline == 1) * BaseDesign: Identifier for the design concept. The same for the initial deisgn and the guideline design for the same design team. * DesignState: currently shown situation. Referenced as S1 - S5 in the paper. * block: Block A or B. See paper * stimuli: Order of stimuli. In each block 108 stimuli were shown. * correct: 1, if response of subject was correct

data <- select (data, VPID, critical, reaction_ms, Guideline, BaseDesign, DesignState, block, stimuli, correct)
head(data)

Analysis of reaction times

Visualize training effect

Visualize training effect. Only for dataset with initial designs (guideline == 0) as an example.

plotdata <- filter(data, Guideline == 0)
plot(plotdata$stimuli, plotdata$reaction_ms)
abline(lm(plotdata$reaction_ms ~ plotdata$stimuli))

summary(lm(plotdata$reaction_ms ~ plotdata$stimuli))

Call:
lm(formula = plotdata$reaction_ms ~ plotdata$stimuli)

Residuals:
   Min     1Q Median     3Q    Max 
-837.1 -362.3 -142.5  204.8 3434.7 

Coefficients:
                  Estimate Std. Error t value Pr(>|t|)    
(Intercept)      1265.7355    24.1820  52.342   <2e-16 ***
plotdata$stimuli   -0.9560     0.3858  -2.478   0.0133 *  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 525.9 on 1942 degrees of freedom
Multiple R-squared:  0.003152,  Adjusted R-squared:  0.002639 
F-statistic: 6.141 on 1 and 1942 DF,  p-value: 0.01329

Effect of guidelines

Fit mixed effects model using maximum likelihood approach. See paper for model argumentation.

data.RT.model <- lmer(reaction_ms ~  Guideline + stimuli + (1|VPID) + (1|BaseDesign), data=data, REML=FALSE)
summary(data.RT.model)

Linear mixed model fit by maximum likelihood  ['lmerMod']
Formula: reaction_ms ~ Guideline + stimuli + (1 | VPID) + (1 | BaseDesign)
   Data: data

     AIC      BIC   logLik deviance df.resid 
 57591.8  57629.4 -28789.9  57579.8     3882 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-2.3552 -0.6038 -0.1606  0.3578  8.4321 

Random effects:
 Groups     Name        Variance Std.Dev.
 VPID       (Intercept)  36558   191.2   
 BaseDesign (Intercept)  20066   141.7   
 Residual               154427   393.0   
Number of obs: 3888, groups:  VPID, 18; BaseDesign, 6

Fixed effects:
             Estimate Std. Error t value
(Intercept) 1271.9955    74.6743  17.034
Guideline   -172.1485    12.6046 -13.658
stimuli       -1.0708     0.2022  -5.296

Correlation of Fixed Effects:
          (Intr) Guidln
Guideline -0.085       
stimuli   -0.148  0.001

Guidelines on average decreased reaction times (-172.1485464 ms)

Fit mixed effects model without guidelines as explanatory variable using maximum likelihood approach

data.RT.nullGuideline <- lmer(reaction_ms ~ stimuli + (1|VPID) + (1|BaseDesign), data=data, REML=FALSE)
summary(data.RT.nullGuideline)

Linear mixed model fit by maximum likelihood  ['lmerMod']
Formula: reaction_ms ~ stimuli + (1 | VPID) + (1 | BaseDesign)
   Data: data

     AIC      BIC   logLik deviance df.resid 
 57772.0  57803.3 -28881.0  57762.0     3883 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-2.5139 -0.6122 -0.1768  0.3610  8.4497 

Random effects:
 Groups     Name        Variance Std.Dev.
 VPID       (Intercept)  36523   191.1   
 BaseDesign (Intercept)  20053   141.6   
 Residual               161880   402.3   
Number of obs: 3888, groups:  VPID, 18; BaseDesign, 6

Fixed effects:
            Estimate Std. Error t value
(Intercept) 1185.766     74.432  15.931
stimuli       -1.068      0.207  -5.159

Correlation of Fixed Effects:
        (Intr)
stimuli -0.152

Use likelihood ratio to estimate significance of guideline usage

data.RT.mGuideline <- anova(data.RT.nullGuideline, data.RT.model)
data.RT.mGuideline

Data: data
Models:
data.RT.nullGuideline: reaction_ms ~ stimuli + (1 | VPID) + (1 | BaseDesign)
data.RT.model: reaction_ms ~ Guideline + stimuli + (1 | VPID) + (1 | BaseDesign)
                      Df   AIC   BIC logLik deviance  Chisq Chi Df Pr(>Chisq)    
data.RT.nullGuideline  5 57772 57803 -28881    57762                             
data.RT.model          6 57592 57629 -28790    57580 182.17      1  < 2.2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Model fit improves with inclusion of guidelines as explanatory variable. Effect is significant (\(\chi^2\)(1)=182.168542, p=NA, 1.629126910^{-41}).

Random slopes for each design

Assumption is, that the effect of the guidelines strongly differs for each design. Adding random slopes to the model:

#fit data
data.RT.modelRandomSlopes <- lmer(reaction_ms ~ Guideline + stimuli + (1|VPID) + (Guideline|BaseDesign), data=data, REML=FALSE)
# print summary
summary(data.RT.modelRandomSlopes)

Linear mixed model fit by maximum likelihood  ['lmerMod']
Formula: reaction_ms ~ Guideline + stimuli + (1 | VPID) + (Guideline |  
    BaseDesign)
   Data: data

     AIC      BIC   logLik deviance df.resid 
 57400.6  57450.8 -28692.3  57384.6     3880 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-2.6441 -0.5761 -0.1534  0.3658  8.8993 

Random effects:
 Groups     Name        Variance Std.Dev. Corr 
 VPID       (Intercept)  36003   189.7         
 BaseDesign (Intercept)  48720   220.7         
            Guideline    32520   180.3    -0.94
 Residual               146250   382.4         
Number of obs: 3888, groups:  VPID, 18; BaseDesign, 6

Fixed effects:
             Estimate Std. Error t value
(Intercept) 1273.7075   101.5412  12.544
Guideline   -172.1505    74.6355  -2.307
stimuli       -1.1022     0.1968  -5.601

Correlation of Fixed Effects:
          (Intr) Guidln
Guideline -0.831       
stimuli   -0.106  0.000

# print parameters (including slopes for designs)
coef(data.RT.modelRandomSlopes)

$VPID
    (Intercept) Guideline   stimuli
L11   1202.8457 -172.1505 -1.102154
L12   1380.4092 -172.1505 -1.102154
L13   1363.8639 -172.1505 -1.102154
L14   1225.6165 -172.1505 -1.102154
L15   1260.8748 -172.1505 -1.102154
L16   1226.6935 -172.1505 -1.102154
L17   1363.5094 -172.1505 -1.102154
L19   1139.4590 -172.1505 -1.102154
L2    1219.1320 -172.1505 -1.102154
L6    1633.8831 -172.1505 -1.102154
L72   1322.0484 -172.1505 -1.102154
L74    786.8722 -172.1505 -1.102154
L75   1096.5575 -172.1505 -1.102154
L76   1513.4307 -172.1505 -1.102154
L77   1430.1906 -172.1505 -1.102154
L78   1399.9991 -172.1505 -1.102154
L79   1339.1572 -172.1505 -1.102154
L8    1022.1921 -172.1505 -1.102154

$BaseDesign
   (Intercept)   Guideline   stimuli
D1    1670.024 -493.553944 -1.102154
D2    1173.032    7.276194 -1.102154
D3    1069.265    6.262199 -1.102154
D4    1122.332 -105.488392 -1.102154
D5    1143.080 -142.968029 -1.102154
D6    1464.511 -304.431321 -1.102154

attr(,"class")
[1] "coef.mer"

# Likelihood ratio test with original model
data.RT.mSlopesGuideline <- anova(data.RT.model, data.RT.modelRandomSlopes)
data.RT.mSlopesGuideline

Data: data
Models:
data.RT.model: reaction_ms ~ Guideline + stimuli + (1 | VPID) + (1 | BaseDesign)
data.RT.modelRandomSlopes: reaction_ms ~ Guideline + stimuli + (1 | VPID) + (Guideline | 
data.RT.modelRandomSlopes:     BaseDesign)
                          Df   AIC   BIC logLik deviance  Chisq Chi Df Pr(>Chisq)    
data.RT.model              6 57592 57629 -28790    57580                             
data.RT.modelRandomSlopes  8 57401 57451 -28692    57385 195.19      2  < 2.2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Adding random slopes for each design significantly improves model fit (\(\chi^2\)(2)=195.1880515, p=NA, 4.125277410^{-43}). The effect of guidelines on reaction time improvement strongly differs between designs

Visualisation of random slopes:

#Mean reaction time values for each design
dSlopes <- ddply(data, .(BaseDesign, Guideline), summarize,  reaction_ms=mean(reaction_ms))
# Plot slope for each design
plot(1,type='n',xlim=c(-0.5,1.5),ylim=c(900,1600),xlab='Guideline usage', ylab='reaction time', xaxt='n')
axis(1,at=seq(0,1), labels=c("initial design", "with guidelines"))

i=1
for (design in unique(dSlopes$BaseDesign)){
    i=i+1
    d <-dSlopes[which(dSlopes$BaseDesign==design), ]
    lines(d$Guideline, d$reaction_ms, col=i, type='o', lwd=2)
}
xrange <- range(dSlopes$Guideline)
yrange <- range(dSlopes$reaction_ms)
legend(1,1600,unique(dSlopes$BaseDesign), col=2:7, lty=1)

# For paper figures: Transform into wide format and use gnuplot
# Gnuplot figure is created by running 'design_slopes_reaction_times.gnuplot' with gnuplot
dSlopes$Guideline <- factor(dSlopes$Guideline)
dSlopesGnuplot <- spread(dSlopes, BaseDesign, reaction_ms)
dSlopesGnuplot$GuidelineLabel <- mapvalues(dSlopesGnuplot$Guideline, from=c('0','1') , to=c('initial','guideline'))
write.csv(dSlopesGnuplot, "design_slopes_reaction_times.csv")

Effect of training

Fit mixed effects model without stimulus order as explanatory variable using maximum likelihood approach. And compare with full model.

data.RT.nullStimuli <- lmer(reaction_ms ~ Guideline + (1|VPID) + (1|BaseDesign), data=data, REML=FALSE)
summary(data.RT.nullStimuli)

Linear mixed model fit by maximum likelihood  ['lmerMod']
Formula: reaction_ms ~ Guideline + (1 | VPID) + (1 | BaseDesign)
   Data: data

     AIC      BIC   logLik deviance df.resid 
 57617.8  57649.1 -28803.9  57607.8     3883 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-2.3626 -0.6083 -0.1687  0.3750  8.4759 

Random effects:
 Groups     Name        Variance Std.Dev.
 VPID       (Intercept)  36550   191.2   
 BaseDesign (Intercept)  19964   141.3   
 Residual               155549   394.4   
Number of obs: 3888, groups:  VPID, 18; BaseDesign, 6

Fixed effects:
            Estimate Std. Error t value
(Intercept)  1213.60      73.74   16.46
Guideline    -172.08      12.65  -13.60

Correlation of Fixed Effects:
          (Intr)
Guideline -0.086

data.RT.mStimuli <- anova(data.RT.nullStimuli, data.RT.model)
data.RT.mStimuli

Data: data
Models:
data.RT.nullStimuli: reaction_ms ~ Guideline + (1 | VPID) + (1 | BaseDesign)
data.RT.model: reaction_ms ~ Guideline + stimuli + (1 | VPID) + (1 | BaseDesign)
                    Df   AIC   BIC logLik deviance Chisq Chi Df Pr(>Chisq)    
data.RT.nullStimuli  5 57618 57649 -28804    57608                            
data.RT.model        6 57592 57629 -28790    57580 27.95      1  1.245e-07 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Model fit improves with inclusion of stimulus order as explanatory variable. Reaction time decreases on average by -1.0707599 ms per stimulus (\(\hat{=}\)-115.6420714 ms reduction from first to last stimulus). This training effect is significant (\(\chi^2\)(1)=27.9503818, p=NA, 1.244663910^{-7}).

Analysis of accuracy of responses

Effect of guidelines

Fit mixed effects model using maximum likelihood approach. See paper for model argumentation.

data.correct.model <- lmer(correct ~ Guideline + (1|VPID) + (1|BaseDesign), data=data, REML=FALSE)
summary(data.correct.model)

Linear mixed model fit by maximum likelihood  ['lmerMod']
Formula: correct ~ Guideline + (1 | VPID) + (1 | BaseDesign)
   Data: data

     AIC      BIC   logLik deviance df.resid 
  -731.6   -700.3    370.8   -741.6     3883 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-4.8040  0.0249  0.1959  0.3673  1.1485 

Random effects:
 Groups     Name        Variance Std.Dev.
 VPID       (Intercept) 0.001435 0.03788 
 BaseDesign (Intercept) 0.001513 0.03890 
 Residual               0.047728 0.21847 
Number of obs: 3888, groups:  VPID, 18; BaseDesign, 6

Fixed effects:
            Estimate Std. Error t value
(Intercept) 0.922840   0.018881   48.88
Guideline   0.046296   0.007007    6.61

Correlation of Fixed Effects:
          (Intr)
Guideline -0.186

Guidelines on average improve accuracy by 4.6296296 % (from 92.2839506 % to 96.9135802 %).

Fit mixed effects model without guidelines as explanatory variable using maximum likelihood approach.

data.correct.nullGuideline <- lmer(correct ~ (1|VPID) + (1|BaseDesign), data=data, REML=FALSE)
summary(data.correct.nullGuideline)

Linear mixed model fit by maximum likelihood  ['lmerMod']
Formula: correct ~ (1 | VPID) + (1 | BaseDesign)
   Data: data

     AIC      BIC   logLik deviance df.resid 
  -690.2   -665.2    349.1   -698.2     3884 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-4.6714  0.0403  0.1809  0.3535  1.0357 

Random effects:
 Groups     Name        Variance Std.Dev.
 VPID       (Intercept) 0.001432 0.03785 
 BaseDesign (Intercept) 0.001512 0.03889 
 Residual               0.048267 0.21970 
Number of obs: 3888, groups:  VPID, 18; BaseDesign, 6

Fixed effects:
            Estimate Std. Error t value
(Intercept)  0.94599    0.01855      51

Use likelihood ratio to estimate significance of guideline usage

data.correct.mGuideline <- anova(data.correct.nullGuideline, data.correct.model)
data.correct.mGuideline

Data: data
Models:
data.correct.nullGuideline: correct ~ (1 | VPID) + (1 | BaseDesign)
data.correct.model: correct ~ Guideline + (1 | VPID) + (1 | BaseDesign)
                           Df     AIC     BIC logLik deviance  Chisq Chi Df Pr(>Chisq)
data.correct.nullGuideline  4 -690.24 -665.18 349.12  -698.24                         
data.correct.model          5 -731.65 -700.32 370.82  -741.65 43.406      1  4.449e-11
                              
data.correct.nullGuideline    
data.correct.model         ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Model fit improves with inclusion of guidelines as explanatory variable. Effect is significant (\(\chi^2\)(1)=43.4057052, p=NA, 4.448891310^{-11}).

Random slopes for each design

Assumption is, that the effect of the guidelines strongly differs for each design. Adding random slopes to the model:

# fit mixed effects model with different accuracy-slopes for each design
data.correct.modelRandomSlopes <- lmer(correct ~ Guideline + (1|VPID) + (Guideline|BaseDesign), data=data, REML=FALSE)
# print summary
summary(data.correct.modelRandomSlopes)

Linear mixed model fit by maximum likelihood  ['lmerMod']
Formula: correct ~ Guideline + (1 | VPID) + (Guideline | BaseDesign)
   Data: data

     AIC      BIC   logLik deviance df.resid 
  -813.0   -769.2    413.5   -827.0     3881 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-4.7327 -0.0197  0.1537  0.3453  1.3859 

Random effects:
 Groups     Name        Variance Std.Dev. Corr 
 VPID       (Intercept) 0.001430 0.03781       
 BaseDesign (Intercept) 0.004498 0.06707       
            Guideline   0.004832 0.06951  -0.90
 Residual               0.046513 0.21567       
Number of obs: 3888, groups:  VPID, 18; BaseDesign, 6

Fixed effects:
            Estimate Std. Error t value
(Intercept)  0.92284    0.02921  31.598
Guideline    0.04630    0.02921   1.585

Correlation of Fixed Effects:
          (Intr)
Guideline -0.848

# print parameters (including slopes for designs)
coef(data.correct.modelRandomSlopes)

$VPID
    (Intercept) Guideline
L11   0.8852863 0.0462963
L12   0.9536868 0.0462963
L13   0.9174748 0.0462963
L14   0.9617339 0.0462963
L15   0.8893099 0.0462963
L16   0.9335690 0.0462963
L17   0.9416161 0.0462963
L19   0.9174748 0.0462963
L2    0.8933334 0.0462963
L6    0.9054041 0.0462963
L72   0.9416161 0.0462963
L74   0.9295454 0.0462963
L75   0.8209094 0.0462963
L76   0.9456397 0.0462963
L77   0.9416161 0.0462963
L78   0.9577103 0.0462963
L79   0.9657575 0.0462963
L8    0.9094276 0.0462963

$BaseDesign
   (Intercept)   Guideline
D1   0.8030276  0.14102449
D2   0.9701533 -0.05125637
D3   0.9977321 -0.01191887
D4   0.9588723  0.02658946
D5   0.8752554  0.11745012
D6   0.9319964  0.05588895

attr(,"class")
[1] "coef.mer"

# Likelihood ratio test with original model
data.correct.mSlopesGuideline <- anova(data.correct.model, data.correct.modelRandomSlopes)
data.correct.mSlopesGuideline

Data: data
Models:
data.correct.model: correct ~ Guideline + (1 | VPID) + (1 | BaseDesign)
data.correct.modelRandomSlopes: correct ~ Guideline + (1 | VPID) + (Guideline | BaseDesign)
                               Df     AIC     BIC logLik deviance  Chisq Chi Df
data.correct.model              5 -731.65 -700.32 370.82  -741.65              
data.correct.modelRandomSlopes  7 -813.02 -769.16 413.51  -827.02 85.375      2
                               Pr(>Chisq)    
data.correct.model                           
data.correct.modelRandomSlopes  < 2.2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Adding random slopes for each design significantly improves model fit (\(\chi^2\)(2)=85.3751098, p=NA, 2.890882610^{-19}). The effect of guidelines on accuracy differs between designs.

Visualisation of random slopes:

# Mean accuracy values for each deisgn converted to percentage
dSlopes <- ddply(data, .(BaseDesign, Guideline), summarize,  accuracy=mean(correct)*100)
# Plot slope for each design
plot(1,type='n',xlim=c(-0.5,1.5),ylim=c(80,100),xlab='Guideline usage', ylab='accuracy', xaxt='n')
axis(1,at=seq(0,1), labels=c("initial design", "with guidelines"))

i=1
for (design in unique(dSlopes$BaseDesign)){
    i=i+1
    d <-dSlopes[which(dSlopes$BaseDesign==design), ]
    lines(d$Guideline, d$accuracy, col=i, type='o', lwd=2)
}
xrange <- range(dSlopes$Guideline)
yrange <- range(dSlopes$accuracy)
legend(1.1,100,unique(dSlopes$BaseDesign), col=2:7, lty=1)

# For paper figures: Transform into wide format and use gnuplot
# Gnuplot figure is created by running 'design_slopes_accuracy.gnuplot' with gnuplot
dSlopes$Guideline <- factor(dSlopes$Guideline)
dSlopesGnuplot <- spread(dSlopes, BaseDesign, accuracy)
dSlopesGnuplot$GuidelineLabel <- mapvalues(dSlopesGnuplot$Guideline, from=c('0','1') , to=c('initial','guideline'))
dSlopesGnuplot

write.csv(dSlopesGnuplot, paste(processing_dir, "design_slopes_accuracy.csv", sep=""))

Effect of training

We didn’t expect to observe a training effect for accuracy, because the vast majority of responses is already correct for the initial designs. Room for improvement is small and harder to measure with binary measurement. If we add stimulus order and compare model fit of our model with this extended one, we get:

data.correct.modelStimuli <- lmer(correct ~ Guideline + stimuli + (1|VPID) + (1|BaseDesign), data=data, REML=FALSE)
summary(data.correct.modelStimuli)

Linear mixed model fit by maximum likelihood  ['lmerMod']
Formula: correct ~ Guideline + stimuli + (1 | VPID) + (1 | BaseDesign)
   Data: data

     AIC      BIC   logLik deviance df.resid 
  -730.7   -693.1    371.3   -742.7     3882 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-4.8188  0.0245  0.1935  0.3690  1.1728 

Random effects:
 Groups     Name        Variance Std.Dev.
 VPID       (Intercept) 0.001435 0.03788 
 BaseDesign (Intercept) 0.001515 0.03893 
 Residual               0.047715 0.21844 
Number of obs: 3888, groups:  VPID, 18; BaseDesign, 6

Fixed effects:
             Estimate Std. Error t value
(Intercept) 0.9165800  0.0198587   46.16
Guideline   0.0463036  0.0070064    6.61
stimuli     0.0001148  0.0001124    1.02

Correlation of Fixed Effects:
          (Intr) Guidln
Guideline -0.177       
stimuli   -0.309  0.001

data.correct.mStimuli <- anova(data.correct.model, data.correct.modelStimuli)
data.correct.mStimuli

Data: data
Models:
data.correct.model: correct ~ Guideline + (1 | VPID) + (1 | BaseDesign)
data.correct.modelStimuli: correct ~ Guideline + stimuli + (1 | VPID) + (1 | BaseDesign)
                          Df     AIC     BIC logLik deviance  Chisq Chi Df Pr(>Chisq)
data.correct.model         5 -731.65 -700.32 370.82  -741.65                         
data.correct.modelStimuli  6 -730.69 -693.10 371.35  -742.69 1.0432      1     0.3071

There seems to be a small tendency for a training effect (accuracy improves by 0.0114786 % per stimulus \(\hat{=}\) 1.2396867 % improvement from first to last stimulus), but it is not significant (\(\chi^2\)(1)=1.0431993, p=NA, 0.3070786).

Exploration

Slopes for each situation of each design

Visualize how close the changes of reaction times are for the different situations in each design. Group by designs to give an impression of closeness/distance.

# Mean reaction timess for each situation of each design
dSituationSlopes <- ddply(data, .(BaseDesign, Guideline, DesignState), summarize,  reaction_ms=mean(reaction_ms))
# Select values from initial designs and guideline designs
initialData <- filter(dSituationSlopes, Guideline==0)
guidelineData <- filter(dSituationSlopes, Guideline==1)
# Join both values for each situation of each design and compare them
dSituationSlopes <- merge(initialData, guidelineData, by=c('BaseDesign','DesignState'))
dSituationSlopes$reaction_ms_Slope <- dSituationSlopes$reaction_ms.x - dSituationSlopes$reaction_ms.y
#Exclude irrelevant columns
dSituationSlopes <- select(dSituationSlopes, BaseDesign, DesignState, reaction_ms_Slope)
head(dSituationSlopes)

# Plot distribution of slopes for each design
dSituationSlopes$BaseDesign <- factor(dSituationSlopes$BaseDesign)
plot(dSituationSlopes$BaseDesign, dSituationSlopes$reaction_ms_Slope)

# Figure from paper is created with gnuplot from this data. For gnuplot transform data into wide format and assign numeric IDs to string descriptors of BaseDesign. 
dSituationSlopes$DesignState <- factor(dSituationSlopes$DesignState)
dSituationSlopes <- spread(dSituationSlopes, DesignState, reaction_ms_Slope)
dSituationSlopes$designIndex <- mapvalues(dSituationSlopes$BaseDesign, from=c('D2','D3','D4','D5','D6','D1') , to=c(1,2,3,4,5,6))
#Then write data to file. Run 'design_situation_slopes.gnuplot' manually with gnuplot to recompile figure.
write.csv(dSituationSlopes, "design_situation_slopes.csv", quote = FALSE)